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ABSTRACT 

Content analysis was used to examine the consistency of responses in self-report data 
obtained from a semi-structured conversational interview. It was hypothesized that a 
majority of the seemingly inconsistent responses would prove to be consistent when 
seen from the respondents' points of view. The sample consisted of 215 female 
homicide offenders incarcerated in New York State correctional facilities or on active 
parole supervision in New York City. Results indicate that of the 24 respondents 
initially identified as having inconsistent responses, only one was regarded as 
inconsistent after the content analysis was performed and a coding scheme developed 
to examine the apparent inconsistency from the respondent's point of view. The 
coding scheme developed also predicted homicide-related variables more effectively 
than did the original coding scheme for this sample. Suggestions for improving the 
construction of interview questions are also presented. 

INTRODUCTION 

The use of self-report data has increased in recent years, particularly as a 
method of obtaining sensitive information, such as involvement in criminal activity, druo 
and alcohol abuse, and sexual practices (Alexander, eta!., 1993; Bailey, et al., 1992; " 
Barnea, et al., 1987; Calsyn, et al., 1993; DeMaio, 1984; Dunford & Elliot, 1984; Hse’r, 
et al., 1992, OMaliey, et al., 1983). Self-report data, however, are often retrospective 
and subject to recall and other types of errors, such as under- or non-reporting of 
socially undesirable behavior; therefore, establishing the reliability and validity of such 
data are of great interest to both researchers who produce these type of data and 
policy makers who use these data to make informed decisions. 

In the field of substance use, which relies heavily on self-report data, Hubbard, 
Eckerman, and Rachal (1976) have proposed three major areas of concern in 
assessing the reliability and validity of such data: external validity, which refers to the 
extent to which the self-report data agree with other indicators of behavior that are 
presumed to be more accurate; construct validity, which is the extent to which the self- 
report measures of substance use relate to other measures in theoretically predicted 
ways, and internal validity, which refers to how internally consistent responses are to 
same or similar items as measured within or across time. Most studies involving 
substance use have been primarily concerned with issues surrounding internal validity 
particularly across time (Alexander, et al., 1993; Bailey, et al., 1992; Barnea, et al., 
1987, Hser, et al., 1992; O'Malley, et al., 1983). These studies typically examined the 
amount of drug use reported by respondents over two or more time periods, and 
reported a measure of validity (usually a correlation coefficient), to assess the degree 
to which responses were consistent across time. These studies have generally found 
a high degree of internal validity in self-report measures of drug use. 
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Studies that have focused on other sensitive topics have used similar across-* 
time techniques to establish internal validity (Calsyn, et al., 1993; McElrath, et al., 
1994), and have found self-report data to be consistent and reliable. Other 
researchers have combined across-time assessments of internal validity and within- 
time assessments, by varying setting or method of test administration across time 
(Needle, et al., 1989; Needle, et al., 1983) and have found that these variables had no 
effect on the consistency of responses. Researchers who have concentrated their 
efforts on assessments of external validity, however, have found little correlation 
between external sources of information and self-report data (Huizinga & Elliot, 1986; 
Dunford & Elliot, 1984; Frick, et al., 1994). 

What is particularly striking about the literature available on this topic is the 
almost complete reliance on quantitative data analysis techniques to establish 
evidence of internal consistency. A few reseoicners have suggested ways in which an 
examination of inconsistencies in self-report data could be improved; ways that could 
lend themselves to qualitative data analysis techniques. For example, Bailey, et al. 
(1991) recommend the use of cognitive analyses; focusing on the thought processes 
behind responses. Respondents would be asked to describe the process they went 
through to recall past experiences in order to shed light on what might appear to be 
logical inconsistencies. These authors also suggest that definitions of terms used in 
surveys should be provided and kept consistent across measures to counteract a 
respondent's tendency to change definitions, especially when measures are taken 
across time. Other researchers have also commented on word ambiguity in a 
measurement instrument as a possible source of response inconsistency (Calsyn, et 
al., 1993), and have suggested ways that a respondent's input could be used to clarify 
terms. However, all of these researchers used interviews with closed-ended response 
formats, which, by their very nature, limit the amount of respondent input during the 
interview process. 

The semi-structured conversational interview, which has the potential to explore 
research topics in greater depth, can provide powerful self-report data, especially when 
the potent combination of fixed-alternative items, open-ended items and scale items 
are used (Kerlinger, 1986; Wood, 1994). When inconsistencies arise in these data, 
the analyst can make use of the responses to open-ended questions to help determine 
the nature of the inconsistencies. Using content analysis to examine these open- 
ended questions is the next logical step. Content analysis is a particularly well-suited 
technique for examining inconsistencies in large data sets because it allows an 
examination of a greater number of subjects than would normally be practical in a 
typical case-study framework (Kerlinger, 1986; Krippendorf, 1980). Content analysis is 
a means of organizing qualitative data in which "themes, issues, and recurring motifs 
within a document can be isolated, counted, and interpreted" (Denzin & Lincoln, 1994, 
pg. 358). In content analysis, categories of responses to open-ended questions are 
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explicitly defined and codes are assigned to these categories (Miles & Huberman, 

1984; Strauss & Corbin, 1990). Codes are revised as data analysis progresses until a 
coding scheme emerges that has conceptual and structural order (Miles & Huberman, 
1984). The use of content analysis, combined with narrative analysis (whereby text is 
analyzed from the perspective of the respondent; Manning & Cullum-Swan, 1994) 
provides powerful techniques "to make better estimates of respondents' true intentions, 
beliefs, and attitudes" (Kerlinger, 1986, pg. 443). These techniques have been 
criticized on the grounds that they are inherently subjective and subject to bias. In 
response, researchers involved in content analysis have suggested that the stability of 
coding schemes be assessed by using two or more independent coders (Miles & 
Huberman, 1984; Westbrook, 1994). 

This paper presents the results of a content analysis that was conducted to 
examine the inconsistency of responses in a semi-structured conversational interview. 

It was hypothesized that a majority of the seemingly inconsistent responses would 
prove to be consistent when seen from the point of view of the respondent's own 
perceptions. 



METHOD 



Data Source/Sample 

The original data used in this study were obtained from semi-structured, 
conversational interviews conducted with female homicide offenders incarcerated in 
New York State correctional facilities or on parole supervision in the New York City 
area. Two hundred and fifteen women (72% of the total number of women who were 
either incarcerated or on active parole supervision) were interviewed by seven 
interviewers from the National Development and Research Institutes, Inc. (NDRI) 
between April 1992 and May 1993. Generally, each interview lasted about one to one- 
and-a-half hours. Questions were both open- and closed-ended. The open-ended 
questions allowed respondents to provide details and explanations about the killings, 
their involvement and the involvement of others, and about their experiences with 
drugs, crime, and violence prior to the homicide event. 

In the original study, respondents were asked early in the interview about their 
histories of involvement in a variety of illegal activities. The question was worded: 
"Have you ever participated in the following activities?" and a list of illegal activities 
was presented, including "kill anyone." At a later point in the interview, respondents 
were asked to describe what took place on the day of the homicide for which they 
were convicted. These responses were initially coded to reflect the subject's own 
perception of their role in the homicide. The initial coding scheme consisted of seven 
categories: Not sure, Denies involvement, Accomplice in other crime but not in 
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homicide, Bystander, Conspirator, Perpetrator, Refused/Missing. Of particular interest 
in this study are the women who had responded to the earlier question that they had 
never participated in killing anyone, and yet were coded as an accomplice, conspirator, 
or perpetrator in the homicide. This subset consisted of 34 women, representing 16% 
of the original sample. Definitions of accomplice, conspirator, and perpetrator that 
were originally used to code the homicide narrative from the point of view of the 
respondent appear below: 

Conspirator: Did not personally commit the homicide but did plan 

it (e.g., hired a killrr or went to the scene with the 
perpetrator, know ; ng that a homicide was going to 
take place). 

Perpetrator: Committed the homicide. 



Accomplice: Participated in another crime, but not in the 

homicide. Respondent willingly participated in a 
crime, but didn't know that the crime would end up 
as a homicide (e.g., participated in a drug deal, 
robbery or kidnapping that another participant turned 
into a homicide; gave a weapon to a friend involved 
in a fight that ended up as an unplanned homicide). 

Of these 34 women, 17 were initially coded as accomplices, 14 were coded as 
perpetrators, and 3 were coded as conspirators. When the narratives were examined, 
however, it became apparent that the respondents who were originally classified a? 
accomplices (N=17) actually fell into two groups. One group was made up of women 
who, even though they reported that they did not know a homicide was intended, 
actively participated in circumstances wherein a homicide was likely to occur (N=7). 
Another group was made up of women who, even though they were involved in 
criminal activity, reportedly had no idea that a homicide would occur (N=10). For 
example, one woman in the latter group reported the following story. (Homicide 
narratives were transcribed by the interviewers from respondents' tape-recorded 
accounts. In all of the homicide narratives presented in this paper, the respondents' 
use of language and grammar remain unchanged. Proper nouns, however, have been 
deleted): 



He (the respondent's sister's friend) made a suggestion and we went 
along with it. He set (us) up with (a) plan to rob this place where she 
(the respondent's sister) had just got (a) job at and on our way there, he 
shot the cab driver. We took a cab. My sister was already there. She 
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worked the midnight shift. He shot the cab driver on the way there. The 
car crashed and that was the crime, why I'm here. 

These 10 women were excluded from the analyses because it was determined that 
their response of 'No' to the question, "Did you ever participate in ... kill anyone?" was 
not inconsistent with their narrative description of the homicide. Therefore, the final 
sample for this analysis consisted of 24 women, representing 11% of the original 
sample. These women did not differ significantly from the rest of the sample in terms 
of race, educational experience, whether they were in prison for the first time, whether 
they had tried any drug, and age at time of homicide (see Table 1). 

Description of Methodology 



The homicde narratives of the 24 women in the sample were initially reviewed 
to determine if the stories told from the respondents' point of view would help to 
explain the apparent inconsistency in their responses. It appeared, from an initial 
content analysis, that the respondents were interpreting the verbs 'participate' and 'kill' 
in a very active sense; that is, if the respondent did not see her actions during the 
homicide as actively involved in the immediate cause of death, or did not see the 
death as the result of a "killing", she indicated that she had never participated in kiHng 
anyone. Based on this review, a coding scheme was developed and refined to assess 
a respondent's level of activity in the homicide, from her own point of view: 



Passive: 



The respondent was held responsible for what she f ailed to do, for 
example, failure to take a child to the hospital/doctor or failure to 
stop a homicide that she knew would occur. 



Reactive: The respondent was reacting to immediate lethal violence, or the 

perceived threat of immediate lethal violence from the victim. 

Avoidant: The respondent set up the situation, or participated in setting up 

the situation wherein the homicide was possible or likely to occur, 
but then withdrew from the situation and responsibility for it. 

Active: The respondent actively participated in the killing, or actively killed 

the victim. 



The active responses are the ones that are truly inconsistent. This coding scheme 
was applied to the narrative data by three independent coders in order to establish the 
stability of the coding scheme (Miles & Huberman, 1984; Westbrook, 1994). The 
average agreement in coding responses among the coders was 89%. 
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RESULTS 



Results indicate that, of the 24 respondents, 11 were coded as avoidant (46%), 
8 were coded as reactive (33%), 4 were coded as passive (17%) and one respondent 
was coded as active (4%). An example of an avoidant response follows: 



The night before, I had asked this guy to go to my house and do my 
parents, you know, 'cause my mother had gotten pissed off and kicked 
me out of the house and like that was the last straw ... so I said, That's 
it. You just gotta do them. You just gotta do both of them'. Because if 
my father lives eventually he's gonna find out that it was me and my 
relationship with him is gonna be blown away ... And so I spoke to this 
guy. Asked him would he go into the house when they weren't there, 
wait for them to come home, rip them off, he can get whatever he 
wanted from the house as long as he took care of my parents. He said, 
'Okay, fine.' ... So he went to the house and the door was locked and he 
couldn't get in. So I kind of felt relief because as I started thinking about 
it it was like, well I really don't want to do this. Maybe if he gets there, 
now that he said the door is locked, now I don't have to punk out and tell 
him I really don't want him to do this. We can set it up for another night 
and I can probably stall him and say I changed my mind, we can run 
something else. You know, leave it alone. But he went back to the 
house, okay, and asked me to go back to the house to unlock the door. 
So I did go back to the house but I didn't unlock the door. And so I went 
back to the bar and told him okay, everything is hooked up. Figuring by 
now ne'll be really pissed off at me. He'll never know I punked out you 
know and he'll say, 'Later for her' and leave it alone or catch me 
whenever he caught me to tell me, 'Yo, what you trying to do to me? 

You set me up. I went there and the door was locked', whatever. So I 
really didn't think about it after that and I went out to sleep at my 
girlfrends house like I normally would. Unbeknownst to me, he decided 
to break into the house when the door wasn't open. Figuring that, for 
whatever reason, the door jammed or whatever and that just wasn't good 
enough and he thought it was an oversight. So he went around the 
house to the back door, broke in through another door and then took 
care of business. So I really thought everything was cool. When the 
next morning the next door neighbors' daughter came over and got me 
and said, 'Hey you gotta come to the house. Your parents are dead.' I 
was like, 'What?' You know, that was like the last thing I expected 
because I really thought it was cool and that nothing was going to 
happen. 
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An example of a reactive response follows: 

A friend of mine asked me to watch his apartment for him. He was a driver. 

His roommate came back and an argument started with a neighbor over a TV 
set. I argued with the lady; told her to wait for my friend then talk about the TV 
set. After I got rid of her, we started arguing. He was high. I saw him reach in 
his back pocket - I knew it was a knife. I took it from him, and used it on him. 

An example of a passive response follows: 

My boyfriend, the guy that I was living with because my husband was in jail. 

He fractured her arm [the subject's two year old daughter]; no, dislocated her 
shoulder, that's what happened. I wrapped it up, I didn't take her to the doctor, 
so the infection spread through her body. ..they said that I was neglectful... I 
should have taken her to the doctor. 

As can be seen from an examination of the narratives, none of the avoidant, 
reactive, or passive women regarded themselves as killers because their perception of 
the role they played in their particular crime is different from that of an active 
participant or perpetrator. The first woman sets her parents up to be murdered, but 
then fails to follow through. She seemed genuinely surprised to hear of her parents' 
death. Although she initially wanted her mother dead, she avoided full responsibility 
for the homicide (from her point of view) by not holding up her end of the bargain (i.e., 
not leaving the door unlocked). The second woman sees herself as acting in self- 
defense. She seems to think that if she had not used the knife on the victim, it would 
have been used on her. This woman did not feel active in the crime so much as 
reactive to the situation. The third woman was passive. When her child was injured 
at the hands of another, she failed to seek appropriate medical attention. These three 
different roles, avoidant, reactive and passive, were not part of the original coding 
scheme but they do serve to help us understand why the above responses are not 
inconsistent with the previous response of "No" to the question "Have you ever 
participated in ... kill anyone?". These women were not intentionally misleading the 
interviewer, the inconsistencies they displayed shed light on their point of view The 
information that they provided can certainly be regarded as consistent. 



The only response that was coded as active, and therefore can be considered 
truly inconsistent, was the case of a woman who suffocated her child shortly after the 
child's birth. Although it was clear from the narrative that the woman was suicidal the 
response was coded as active because of the respondent's description of her actions 
that appeared quite deliberate. 
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Further analyses were conducted to compare level of activity as indicated by 
the newly created coding scheme (avoidant, passive, reactive, and active) and role in 
homicide as indicated in the original coding scheme (accomplice, conspirator, and 
perpetrator) for this sample. Both coding schemes were used to examine the 
relationship of role in homicide and level of activity with variables associated with the 
homicide, such as motive for homicide, method used in homicide, and whether the 
respondent believed the victim did anything to bring about his or her own death. An 
examination of Tables 2-4 reveals that none of these homicide factors varied as a 
function of role, but did vary as a function cf level of activity. Reactive respondents 
were much more likely to kill in self-defense or in defense of others (88%), while 
avoidant respondents committed homicide more often because of jealousy or revenge 
(28%), or in order to obtain money and/or drugs (27%). In terms of method use ' to 
kill, level of activity was used to examine the use of guns, knives, or other methods. 
Reactive respondents were more likely to use a knife (75%), while the other 
respondents tended to use other methods, such as blunt instruments, strangulation, or 
by abuse or neglect. When respondents were asked if the victim did anything to bring 
about their own death, reactive respondents were overwhelmingly likely to respond 
'Yes’ (88%), while only 30% of avoidant respondents and none of the passive or active 
respondents replied 'Yes'. 



DISCUSSION 

Previous research on examining consistency of response in self-report data fails 
to take into consideration the perspective of the interview subject. Due to the many 
facets of human nature, responses that on the surface appear to be inconsistent may 
very well prove, on further examination, to reflect a consistent world view. This 
research suggests a qualitative method for exploring these inconsistent responses that 
takes the subject's perceptions into account. The research presented illustrates the 
benefits of utilizing content analysis as a strategy for examining inconsistencies in self- 
report data. The results suggest that the coding scheme which resulted from the 
content analysis not only served to reduce the number of truly inconsistent responses, 
(thereby increasing the usable data set), but also functions as a better predictor of 
homicide-related variables for these respondents than did the original coding scheme. 

This research also has implications for item construction in semi-structured 
conversational interviews. It is recommended that future studies that require 
respondents to answer sensitive questions, such as involvement in homicide, use 
language that is less open to subjective interpretation by the respondents. For 
example, instead of asking the question, 'Did you ever participate in ... kill anyone?' 
the interview schedule could include the following questions: 

Have you ever deliberately and directly caused someone else's death? 
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Have you ever accidently but directly caused someone else's death? 

These questions can be repeated to probe for indirect involvement in the cause of 
death, such as in the neglect cases, or in the conspiracy cases. While it may seem 
burdensome to ask so many questions, it has been our experience that respondents 
can tolerate being asked similar types of questions when interviewers are properly 
tiained to vary voice, inflection and posture. 

It is clear from the results of this research that content and narrative analyses 
are useful methodologies for examining the inconsistencies of responses in self report 
data. It is hoped that future research will examine the generalizability of these 
methodologies to similar types of problems in self report data. For example, content 
analysis might also be used to examine construct validity, if the operational definition 
of the construct in the validating measure is used in conjunction with the content 
analysis in developing the coding scheme. 
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Table 1 



Demographic Comparison of Sub Sample (N=24) with Balance of Sample (N=191) 



Race/Ethnicity 


Sub Sample 


Balance of Sample 


Black 


42% 


58% 


White 


38% 


21% 


Hispanic 


12% 


14% 


Other 


8% 


7% 



f = 3. 9, (3), p=. 272 



Education 


Sub Sample 


Balance of Sample 


Jr high or less 


9% 


8% 


Jr High - High School 


82% 


87% 


More than High School 


9% 


5% 



X 2 = 4. 6,(2), p=. 793 





Sub Sample 


Balance of Sample 


First time in prison? 

(percent who 
responded "Yes")* 


100% 


93% 


Ever use any drug? 

(percent who 
responded "Yes")** 


96% 


95% 


Age at time of 
homicide*** 


30.9 


27.1 



*t = 1 8,(1 ),p=.1 84; **x 2 = -014,(1 ),p=.905; ***t=1 .8,(136), p=.074 
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Motive by Role in Homicide and Level of Activity 



Active 


0% 


0% 


0% 


0% 


0% 


1 00% 


0% 


Avoidant 


0% 


13% 


27% 


28% 


9% 


0% 


18% 


Reactive 


0% 


88% 


0% 


0% 


12% 


0% 


0% 


Passive 


25% 


0% 


25% 


0% 


0% 


50% 


0% 


Perpetrator 


7% 


43% 


14% 


0% 


14% 


22% 


0% 


Conspirator 


0% 


CO 

CO 


-o 

CO 

CO 


34% 


0% 


0% 


0% 


Accomplice 


0% 


29% 


14% 


29% 


0% 


0% 


28% 




No Motive 


Seif- 

Defense/Defense 
of Others 


Get money/ 
Drugs 


Jealousy/Revenge 


Discipline/Scare 

Tactic 


Other 


Missing 



II 



o 

ERLC 



14.2, (12), P=.287 y 2 = 35.2, (18), P=.009 



Method by Role in Homicide and Level of Activity 



Active 


0% 


0% 


100% 


Avoidant 


9% 


0% 


91% 


Reactive 


13% 


75% 


12% 


Passive 


25% 


0% 


75% 


Perpetrator 


7% 


36% 


57% 


Conspirator 


33% 


0% 


67% 


Accomplice 


14% 


14% 


72% 




Gun 


Knife 


Other 



N* 

O 

O 

if 

CL 



CD 



N- 



II 

CM 



CD 

LO 

if 

CL 

? 

CD 

CM 

CO 

II 

:m 




00 







.286, (2), P=.87 x 2 = 9.07, (2), P=,010 



